---
id: conversation-simulator
title: Conversation Simulator
sidebar_label: Conversation Simulator
---

<head>
  <link
    rel="canonical"
    href="https://deepeval.com/docs/conversation-simulator"
  />
</head>

`deepeval`'s `ConversationSimulator` allows you to simulate full conversations between a fake user and your chatbot, unlike the [synthesizer](/docs/synthesizer-introduction) which generates regular goldens representing single, atomic LLM interactions.

```python title="main.py" showLineNumbers
from deepeval.test_case import Turn
from deepeval.simulator import ConversationSimulator
from deepeval.dataset import ConversationalGolden

# Create ConversationalGolden
conversation_golden = ConversationalGolden(
    scenario="Andy Byron wants to purchase a VIP ticket to a cold play concert.",
    expected_outcome="Successful purchase of a ticket.",
    user_description="Andy Byron is the CEO of Astronomer.",
)

# Define chatbot callback
async def chatbot_callback(input):
    return Turn(role="assistant", content=f"Chatbot response to: {input}")

# Run Simulation
simulator = ConversationSimulator(model_callback=chatbot_callback)
conversational_test_cases = simulator.simulate(conversational_goldens=[conversation_golden])
print(conversational_test_cases)
```

The `ConversationSimulator` uses the scenario and user description from a `ConversationalGolden` to simulate back-and-forth exchanges with your chatbot. The resulting dialogue is used to create `ConversationalTestCase`s for evaluation using `deepeval`'s multi-turn metrics.

## Create Your First Simulator

To create a `ConversationSimulator`, you'll need to define a callback that wraps around your LLM chatbot.

```python
from deepeval.test_case import Turn
from deepeval.simulator import ConversationSimulator

async def model_callback(input: str, turns: List[Turn], thread_id: str) -> Turn:
    return Turn(role="assistant", content=f"I don't know how to answer this: {input}")

simulator = ConversationSimulator(model_callback=model_callback)
```

There are **ONE** mandatory and **FOUR** optional parameters when creating a `ConversationSimulator`:

- `model_callback`: a callback that wraps around your conversational agent.
- [Optional] `simulator_model`: a string specifying which of OpenAI's GPT models to use for generation, **OR** [any custom LLM model](/docs/metrics-introduction#using-a-custom-llm) of type `DeepEvalBaseLLM`. Defaulted to `gpt-4.1`.
- [Optional] `async_mode`: a boolean which when set to `True`, enables **concurrent simulation of conversations**. Defaulted to `True`.
- [Optional] `max_concurrent`: an integer that determines the maximum number of conversations that can be generated in parallel at any point in time. You can decrease this value if you're running into rate limit errors. Defaulted to `100`.

### Model callback

Only the `input` argument is required when defining your `model_callback`, but you may also define these optional arguments:

- [Optional] `turns`: a list of `Turn`s, which include the role and content of each message in the conversation.
- [Optional] `thread_id`: a unique identifier for each conversation.

While turns captures the dialogue context for each turn, some applications must persist additional state across turns — for example, when invoking external APIs or tracking user-specific data. In these cases, you'll want to take advantage of the `thread_id`.

```python title="main.py"
from deepeval.test_case import Turn

async def model_callback(input: str, turns: List[Turn], thread_id: str) -> Turn:

    # Inspect the turns and thread_id
    print(turns)
    print(thread_id)

    # Replace with your chatbot
    res = await your_llm_app(input, turns, thread_id)
    return Turn(role="assistant", content=res)
```

## Simulate A Conversation

To simulate your first conversation, simply pass in a list of `ConversationalGolden`s to the `simulate` method:

```python
from deepeval.dataset import ConversationalGolden
...

conversation_golden = ConversationalGolden(
    scenario="Andy Byron wants to purchase a VIP ticket to a cold play concert.",
    expected_outcome="Successful purchase of a ticket.",
    user_description="Andy Byron is the CEO of Astronomer.",
)
conversational_test_cases = simulator.simulate(conversational_goldens=[conversation_golden])
```

There are **ONE** mandatory and **ONE** optional parameter when calling the `simulate` method:

- `conversational_goldens`: a list of `ConversationalGolden`s that specify the scenario and user description.
- [Optional] `max_user_simulations`: an integer that specifies the maximum number of user-assistant message cycles to simulate per conversation. Defaulted to `10`.

A simulation ends either when the converaiton achieves the expected outcome outlined in a `ConversationalGolden`, or when the `max_user_simulations` has been reached.

:::tip
You can also generate conversations from existing turns. Simply populate your `ConversationalGolden` with a list of initial `Turn`s, and the simulator will continue the conversation.
:::

## Advanced Usage

### Lifecycle hooks

The `ConversationSimulator` provides a `on_simulation_complete` hooks that allow you to execute custom logic whenever a simulation of an individual test case has completed. This allows you to process each `ConversationalTestCase` as soon as it's generated, rather than waiting for all simulations to finish.

```python
from deepeval.simulator import ConversationSimulator
from deepeval.test_case import ConversationalTestCase

def handle_simulation_complete(test_case: ConversationalTestCase, index: int):
    print(f"Conversation {index} completed with {len(test_case.turns)} turns")

conversational_test_cases = simulator.simulate(
    conversational_goldens=[golden1, golden2, golden3],
    on_simulation_complete=handle_simulation_complete
)
```

The hook function receives two parameters:

- `test_case`: the completed `ConversationalTestCase` object containing all turns and metadata.
- `index`: the index of the corresponding golden that was simulated (**ording is perserved** during simulation).

:::tip
When using `async_mode=True`, conversations may complete in any order due to concurrent execution. Use the `index` parameter to track which golden each test case corresponds to.
:::

### Existing turns

If your multi-turn chatbot has one or more predefined turns (for example, a hardcoded assistant message at the beginning of a conversation), you would simply include this as part of the simulation by providing a list of preexisting `turns` to a `ConversationalGolden`:

```python
from deepeval.test_case import ConversationalTestCase, Turn

golden = ConversationalGolden(turns=[Turn(role="assistant", content="Hi! How can I help you today?")])
```

By including a list of non-empty `turns`, `deepeval` will run simulations based on the additional context you've provided.

## Evaluate Simulated Turns

The `simulate` function returns a list of `ConversationalTestCase`s, which can be used to evaluate your LLM chatbot using `deepeval`'s conversational metrics. Use simulated conversations to run [end-to-end](/docs/evaluation-end-to-end-llm-evals) evaluations:

```python
from deepeval import evaluate
from deepeval.metrics import TurnRelevancyMetric
...

evaluate(test_cases=conversational_test_cases, metrics=[TurnRelevancyMetric()])
```
